Empirically Valid Rules for Ill-Defined Domains

Ill-defined domains such as writing and design pose challenges for automatic assessment and feedback. There is little agreement about the standards for assessing student work nor are there clear domain principles that can be used for automatic feedback and guidance. While researchers have shown some success with automatic guidance through apriori rules and weak-theory structuring these methods are not guaranteed widespread acceptance nor is it clear that the lessons will transfer out of the tutoring context into realworld practice. In this paper we report on data mining work designed to empirically validate a-priori rules with an exploratory dataset in the domain of argument diagramming and scientific writing. We show that it is possible to identify diagram rules that correlate with student performance but that direct correlations can often run counter to expert assumptions and thus require deeper analysis.